Libraries tagged by Robots Exclusion Protocol
vipnytt/robotstxtparser
25 Favers
553381 Downloads
553381 Downloads
Robots.txt parsing library, with full support for every directive and specification.
bnomei/kirby3-robots-txt
19 Favers
32240 Downloads
32240 Downloads
Manage the robots.txt from the Kirby config file
crwlr/robots-txt
9 Favers
5455 Downloads
5455 Downloads
Robots Exclusion Standard/Protocol Parser for Web Crawling/Scraping
vipnytt/useragentparser
2 Favers
707038 Downloads
707038 Downloads
User-Agent parser for robot rule sets
diggin/diggin-robotrules
7 Favers
184 Downloads
184 Downloads
parser/handler for Robots Exclusion Protocol (robots.txt and more)